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The  Institute  for  Defense  Analyses  is  a  non-profit  corporation  that  operates 
three  federally  funded  research  and  development  centers  to  provide  objective 
analyses  of  national  security  issues,  particularly  those  requiring  scientific  and 
technical  expertise,  and  conduct  related  research  on  other  national  challenges. 
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Introduction 

Unexploded  Ordnance  (UXO)  may 

contaminate  1400  sites  in  over  10M 
acres  in  the  United  States*: 

o  UXO  can  bury  in  the  ground  and  become 
difficult  to  distinguish  from  metallic  clutter 
using  traditional  detection  methods 

o  As  recently  as  the  early  2000s,  75%  of 
remediation  costs  were  spent  on  false 
alarms  -  excavating  objects  that  turned 
out  to  be  clutter 

*  Defense  Science  Board  (DSB).  Report  of  the  Defense  Science  Board  Task  Force  of  Unexploded  Ordnance.  December  2003 
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UXO 

Clutter 

The  Environmental  Security  Technology 
Certification  Program  (ESTCP)  has 
sponsored  the  development  of  advanced 
geophysical  instruments  and  software 
to  rank  the  likelihood  that  a  buried 
object  is  UXO: 

o  Objects  with  a  low  UXO  likelihood  can  be 
classified  as  clutter  and  left  unexcavated, 
reducing  remediation  costs 

o  Remediation  resources  can  then  be 
directed  towards  excavating  all  other 
objects,  allowing  more  land  to  be  cleared 
for  the  same  price 

ESTCP  has  sponsored  a  series  of 
demonstrations  to  assess  the  capability 
of  advanced  geophysical  classification 
technologies  for  safely  remediating  land 
of  UXO  while  reducing  remediation  costs 
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Methods 


For  each  demonstration 


ESTCP  emplaced  inert  and  surrogate  UXO  in  the 
ground  to  increase  the  number  of  Targets  of 
Interest  (TOI): 

o  TOI  =  UXO,  inert  and  surrogate  munitions, 
seeds,  and  any  other  objects  for  which  the 
site  team  agrees  that  removal  is  required 

Thousands  of  clutter  objects  were  already  native 
to  the  site 


TOI  emplacement 
plan  for  the  Camp  San 
Luis  Obispo  (SLO) 
demonstration: 

- Site  Boundary 

- Road  or  Rocks 

o  TOI  emplacement 


Ranked  Anomaly  List 


Collecting  data  with 
the  Time-domain 
ElectroMagnetic  Towed 
Array  Detection  System 
(TEMTADS) 


Target  ID 

Category 

Dig  Decision  | 

XX-2003 

Cannot  Analyze 

Excavate 

XX-2013 

Cannot  Analyze 

Excavate 

XX-2030 

Cannot  Analyze 

Excavate 

XX-2323 

Training  Set 

Excavate 

XX-2338 

Training  Set 

Excavate 

XX-2413 

Likely  TOI 

Excavate 

XX-2504 

Likely  TOI 

Excavate 

XX-2665 

Likely  TOI 

Excavate 

XX-2503 

Likely  TOI 

Excavate 

XX-2278 

Likely  TOI 

Excavate 

XX-2677 

Cannot  Decide 

Excavate 

XX-2364 

Cannot  Decide 

Excavate 

XX-2616 

Cannot  Decide 

Excavate 

_ L 

XX-2674 

Likely  Clutter 

Leave  Unexcavated 

XX-2058 

Likely  Clutter 

Leave  Unexcavated 

XX-2274 

Likely  Clutter 

Leave  Unexcavated 

XX-2054 

Likely  Clutter 

Leave  Unexcavated 

XX-2541 

Likely  Clutter 

Leave  Unexcavated 
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Demonstrators  collected  data  using 
advanced  instruments  and  processed  the 
data  using  advanced  software  to  produce  a 
ranked  anomaly  list 

Each  ranked  anomaly  list  consisted  of  an 
ordered  set  of  locations  detected  at  the  site, 
ranked  by  their  likelihood  of  containing  a 
buried  TOI 

Demonstrators  selected  a  dig  threshold  to 
apply  to  each  ranked  anomaly  list 


ROC  Curve 


ESTCP  excavated  all  detected  locations  to 
construct  ground  truth 

The  Institute  for  Defense  Analyses  (IDA)  compared 
each  list  to  ground  truth  and  created  a  Receiver- 
Operating  Characteristic  (ROC)  curve  to  score  the 
list's  performance 
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Retrospective 
best  case  scenario 

By  definition: 
100%  TOI 

correctly  classified 
with  best 
clutter  rejection 

(Training  Set 
too  small  to  see) 


Likely  Clutter 


Dig  threshold 

For  this  list: 
100%  TOI 

correctly  classified 
(to  the  right 
of  open  dot) 

For  other  lists: 
Less  than  100%  TOI 
correctly  classified 
(down  &  left 
of  open  dot) 


Excavate 

Everything 


Leave  Everything 
Unexcavated 


0  100  200  300  400  500 

Number  of  True  Clutter 
Incorrectly  Classified 
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Results 


UXO  Live  Site  Demonstrations,  FY2007  -  FY2017 


2007 

Sibert 

Camp  Sibert,  AL 

42 

4.2in 

Berkeley  UXO  Discriminator  (BUD),  EM61-MI<2  Cart,  EM63  (cued),  GEM-3  (cued),  Multi-Sensor  Towed  Array  Detection  System  (MTADS)  -  EM61-Mk2/GEM-3/mag 

2009 

SLO 

Camp  San  Luis  Obispo,  CA 

59 

37mm,  60mm,  81mm,  2.36in,  3in,  4.2in,  5in 

BUD,  EM61-Mk2  Cart,  MTADS  -  EM61-Mk2/mag,  Man-portable  Simultaneous  magnetometer  and  ElectroMagnetic  System  (MSEMS),  MetalMapper,  Time-domain 
ElectroMagnetic  Towed  Array  Detection  System  (TEMTADS)  5x5 

2010 

Butner 

Camp  Butner,  NC 

54 

37mm,  M48fuze,  105mm 

EM61-MI<2  Cart,  MetalMapper,  TEMTADS  5x5 

2011 

PoleMtn 

Pole  Mountain,  WY 

36 

small  Industry  Standard  Object  (ISO),  37mm,  57mm,  60mm,  75mm,  3in 

EM61-MI<2  Cart,  MetalMapper 

Beale 

Camp  Beale,  CA 

95 

small  ISO,  37mm,  60mm,  81mm,  105mm 

BUD,  EM61-Mk2  Cart,  MetalMapper,  Man  Portable  Vector  (MPV),  TEMTADS  2x2 

2012 


FtSill  Fort  Sill,  OK  9 

MMR1  Massachusetts  Military  Reservation,  MA  24 

Spencer  Camp  Spencer,  TN  75 


signal  flare,  RA/RI  pipe,  Laws  rocket  motor,  grenade,  rifle  grenade, 
practice  grenade,  flare  canister,  small  ISO,  37mm,  40mm,  2.36in,  3.5in 

parachute  flare,  illumination  round,  smoke  canister,  cartridge,  37mm, 
medium  ISO,  60mm,  2.5in,  75mm,  81mm,  105mm,  4.2in,  155mm,  8in 

small  ISO,  37mm,  medium  ISO,  60mm,  75mm,  3in,  105mm,  155mm 


MetalMapper 

MetalMapper,  TEMTADS  2x2 
MetalMapper,  MPV,  TEMTADS  2x2  and  5x5 


GeorgeWest 

Camp  George  West,  CO 

5 

medium  ISO,  75mm 

MPV 

small  ISO,  hand  grenade,  rifle  grenade,  2.36in,  2.36in  motor,  3.5in 

RuckerPoP 

Fort  Rucker,  AL 

32 

motor,  one  2.36in  motor  &  one  3.5in  motor,  two  2.36in,  two  2.36in 
motors,  three  2.36in  motors,  three  2.36in  motors  &  one  3.5in  motor, 
one  2.36in  &  118  2.36in  motors 

MetalMapper,  TEMTADS  2x2  and  5x5 

2013  CM- 

Ellis 

Camp  Ellis,  IL 

18 

small  ISO,  grenade,  hand  grenade,  rifle  grenade,  2.36in 

MetalMapper,  TEMTADS  2x2 

SWPG 

Southwestern  Proving  Ground,  AR 

21 

20mm,  small  ISO,  37mm,  40mm,  medium  ISO,  75mm,  90mm,  105mm 

MetalMapper,  TEMTADS  2x2 

l/2in  x  2in  pipe,  l/2in  x  3in  pipe,  20mm,  two  20mm,  small  ISO,  scar 

NewBoston 

New  Boston  Air  Force  Station,  NH 

9 

rocket,  382  practice  bomb,  M103  bomb  nose  fuze,  1001b  bomb,  Ml 
practice  bomb 

MPV,  TEMTADS  2x2 

2014 


Waikoloa 


FtBliss 


Waikoloa,  FH I 

Castner  Range  in  Fort  Bliss,  TX 


9 

12 


small  ISO,  37mm,  medium  ISO,  60mm,  81mm,  TOI  frag 
25mm,  small  ISO,  37mm,  105mm 


MetalMapper,  MPV,  MPV3D 
TEMTADS  2x2 


From  FY2007  -  FY2017, 
IDA  scored 
538  lists  created  by 
22  organizations  using 
11  instruments  in 
19  demonstrations 


Guam 

Andersen  Air  Force  Base,  Guam 

14 

small  ISO,  37mm 

TEMTADS  2x2 

2015 

FtOrd 

Fort  Ord,  CA 

4 

20mm,  35mm,  small  ISO,  37mm,  40mm,  57mm,  medium  ISO,  60mm, 
75mm,  81mm,  large  ISO,  105mm,  4.2in,  155mm 

MetalMapper 

2016 

SWPG2 

Southwestern  Proving  Ground,  AR 

8 

20mm,  small  ISO,  37mm,  40mm,  57mm,  75mm,  90mm 

One-Pass  Time-domain  ElectroMagnetic  induction  Array  (OPTEMA),  TEMTADS  2x2 

2017 

Tobyhanna 

Tobyhanna  Army  Depot,  PA 

2 

small  ISO,  medium  ISO,  75mm 

MPV 

Performance  of  Every  Final  List  from  Every  Demonstration,  FY2007  -  FY2017 


IF  100%  of  TOI  were  correctly  classified  at  the  dig  threshold, 
THEN  plot  clutter  rejection  at  the  dig  threshold, 
with  filled  markers  indicating  who  created  the  list: 

•  Technology  Developer 
A  Production-Level  Remediation  Company 


ELSE  IF  less  than  100%  of  TOI  were  correctly  classified  at  the  dig  threshold, 
THEN  plot  clutter  rejection  for  the  retrospective  best  case  scenario, 
with  open  markers  indicating  who  created  the  list: 

O  Technology  Developer 
A  Production-Level  Remediation  Company 
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Discussion 

By  design,  later  demonstrations  typically 
included  smaller  TOIs: 

o  Rule  of  thumb:  the  smaller  the  TOI, 
the  more  difficult  the 
demonstration 

Overall,  demonstration  results  typically 
improved  with  chronological  order: 

o  By  2011,  most  organizations  at 
most  sites  could  correctly  classify 
100%  TOI  while  rejecting  more 
than  70%  clutter 

o  On  other  sites,  advanced 
geophysical  classification 
demonstrated  utility  for  more 
directed  goals,  e.g.: 

-  FtOrd:  large,  shallow  munitions  could 
be  correctly  classified 

-  MMR1:  large  munitions  with 
unexpended  explosive  material  could 
be  correctly  classified 

Improvements  in  classification  results 
over  time  were  likely  due  to  advances  in 
instruments,  software,  and  processes 
for  quality  control  and  quality  assurance 

Advanced  geophysical  classification 
technology  now  has  the  capability  to 
focus  remediation  efforts  on  the 
excavation  of  explosive  hazards  while 
reducing  the  costly  and  unnecessary 
excavation  of  non-explosive  clutter 

Sponsored  by  the  Environmental  Security  Technology  Certification  Program  (ESTCP) 
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